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Abstract 

In the United States, tests are held to high standards of quality. In developing 
countries such as Malawi, psychometricians must deal with these same high 
standards as well as several additional pressures such as widespread 
cheating, test administration difficulties due to challenging landscapes and poor 
resources, difficulties in reliably scoring performance assessments, and 
extreme scrutiny from political parties and the popular press. The purposes of 
this paper are to (a) familiarize the measurement community in the US about 
Malawi’s assessment programs, (b) discuss some of the unique challenges 
inherent in such a program, (c) compare testing conditions and test 
administration formats between Malawi and the US, and (d) provide 
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suggestions for improving iarge-scaie testing in countries such as the US and 
Maiawi. By iearning how a smaii country instituted and supports its current 
testing programs, a broader perspective on resoiving current measurement 
probiems throughout the worid wiii emerge. 


Maiawi is a smaii iandiocked country in Africa, south of the Equator covering an area of 118, 484 
square kiiometers of which 20% is water. The country is bordered to the North and North-East 
by the Repubiic of Tanzania and to the East, South and South-West by the Repubiic of 
Mozambique. The Repubiic of Zambia forms the Western border. 

Maiawi gained independence from Britain in 1964 and operated under one-party state untii 1994 
when a muitiparty government was eiected. The popuiation of Maiawi is estimated at 1 1 miiiion 
peopie. About 46% of the popuiation consists of chiidren and youth iess than 15 years of age. 

The iiteracy ievei is estimated at 40% of the aduit popuiation (29% femaie and 48% maie). 
Because of poor ieveis of iiteracy, there has been rampant poverty. This situation prompted the 
new government to introduce Free Primary Education (FPE) in 1994 as a tooi for aiieviating 
poverty. This innovation received an overwheiming support from the pubiic in that the enroiment 
in primary schoois increased from 1.8 miiiion to about 3 miiiion pupiis. This meant an increased 
demand for resources that support iearning, inciuding assessment. 

In this paper, we provide an overview of the nationai testing systems that support FPE in 
Maiawi. The psychometric, iogistic, and poiiticai factors affecting this system are discussed, as 
are the simiiarities and differences between educationai testing in Maiawi and in the United 
States. We begin with a description of the Maiawi Nationai Examinations Board. 

A Brief History of the Maiawi Nationai Examinations Board 

In 1969, the Malawi parliament enacted a law that created the Malawi Certificate Examination 
Board (MCE Board). This Board was charged with the responsibility of developing and 
administering the Malawi Certificate of Education (MCE) examination in conjunction with the 
Associated Examining Board (AEB) of the UK. The first such examination was administered in 
1972. Prior to 1972, school leavers in Malawi were taking the Cambridge Overseas School 
Leaving Examination from the UK. 

Seven years later, the MCE Board became the Malawi Certificate Examinations and Testing 
Board (MCE and TB). The MCE and TB continued to administer the MCE examinations with the 
AEB until 1989 when the handover was completed. 

Following an evaluation of examinations in Malawi in 1984, it was decided that all public 
examinations should be developed and administered by one central authority. Consequently, in 
1987, parliament approved legislation merging the examinations section of the Ministry of 
Education with the MCE and TB, thus forming the Malawi National Examinations Board 
(MANEB), which currently operates the major educational testing programs in Malawi. In 
addition, MANEB took over the responsibility of developing and administering Teacher 
Certificate Examinations and Craft Examinations for technical schools. 

Malawi’s Education System 

The Malawi education system consists of three levels: primary, secondary, and tertiary. The 
primary education level is an eight-year cycle running from Standards (grades) 1 to 8. Standard 
8 is an equivalent of Grade 8 in the United States. At the end of Standard 8 pupils take the 
Primary School Leaving Certificate Examination (PSLCE). 

Secondary education lasts for four years, running from Form 1 to Form 4, which are the 
equivalents of Grades 9 to 12 in the U.S. Two national examinations are administered at this 
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level: the Junior Certificate Examination (JCE) at the end of junior secondary In Form 2 and the 
Malawi School Certificate Examination (MSCE) at the end of senior secondary In Form 4. 

Tertiary education Is usually four years particularly at the University level, although there are 
other tertiary educational Institutions that offer courses and programs for less than four years. 
Teacher training for primary school teachers Is usually two years, while technical training may 
last for four years or less depending on the field of specialization. Access to tertiary education Is 
still very limited because of scarcity of places at that level. 

A description of these major educational testing programs follows. Table 1 presents a brief 
summary of these programs that Includes the grade at which they are administered, the purpose 
of the test, as well as the number of students sitting for each test and the passing percentages 
for the most recent administration on which data are available. 


Table 1 

Summary of Malawi’s Major Educational Assessments 

Exam 

Grade 

Administered 

Purpose 

# Examinees 
2001 

% Pass 
2001 

PSLCE 

8 

secondary school entrance 

161,786 

26.33* 

JCE 

10 

10th grade exit; basic 
employment certificate 

82,530 

57.21 

MSCE 

12 

High school exit; postsecondary 
admissions 

61,856 

18.01 


*This represents proportion of PSLCE examinees selected to secondary school. 

Major Testing Programs in Malawi 

MANEB develops and administers three major national school examinations: PSLCE, JCE and 
MSCE. A brief description of these examination programs is provided below. 

The PSLCE 

PSLCE terminates the primary cycle. Its results are used for certification and selection into Form 
1 of the secondary education. The results are reported in letter grades A-F, where A denotes 
excellent performance and F a fail. Five subjects are offered at this level. These are English, 
Mathematics, Primary Science, Chichewa (a local language), and Social Studies. For selection 
purposes, students are ranked within their districts. Each district is allocated a certain number of 
Form 1 places in national secondary schools. The district quota depends on the proportion of 
candidates in the district in relation to the national total. The remaining candidates are 
considered for places in District Secondary Schools and Community Day Secondary Schools 
(CDSSs). At each selection level, boys and girls are considered separately to ensure gender 
equity (i.e., within-group norming). A single merit list would result in boys getting a 
disproportionate number of secondary school places, since they generally perform better than 
girls. 

For many people, the certification aspect of the PSLCE is not as important as its selection 
function, because the certificate can no longer be used for employment purposes as is the case 
with MSCE. Therefore the pupils are under pressure to perform well enough to be selected into 
secondary education. A longitudinal sampling of the numbers of students taking the PSLCE and 
the numbers of students passing it, are presented in Table 2. As these data show, the demand 
for secondary education has always outstripped the available places. 

I 1 
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Table 2 

Standard 8 - Form 1 Transition 


Year 

PSLCE Entry 

# Passing 

% Selected 

1977 

47,317 

4,854 

10.3 

1987 

95,631 

6,894 

7.2 

1997 

128,379 

9,170 

7.1 

2001 

161,786 

42,600 

*26.3 


*CDDSs were instituted in 1 998. This figure inciudes the 5% of students who went to nationai 
secondary schoois and the 21% who went to CDSS’s. 

Formai education for those who faii to get into nationai secondary schooi effectiveiy stops at 
Standard 8. Before 1999, some pupiis received secondary schooi tuition through Distance 
Education Centers (DECs), which have since been turned into Community Day Secondary 
Schoois (CDSS). The increased transition rate in 2001 is a reflection of the inciusion of students 
who go to CDSSs. However, the quaiity of iearning at CDSSs is considered inferior to the 
conventionai secondary schoois in terms of materiais and number and quaiity of teachers in 
these schoois. Private secondary schoois aiso provide secondary education but are too 
expensive for most parents, and the majority of them are not weil resourced. Untii now the 
competition is high for pieces in the conventionai secondary schoois where the government 
subsidizes tuition and the schoois are better resourced than the private and CDSSs. This is 
what makes PSLCE a high-stakes examination: it determines one’s opportunity for higher, better 
and affordabie education. 

JCE 

The JCE is administered after two years of secondary education. Originaiiy this examination was 
meant to assess skiiis and knowiedge ieading to gainfui empioyment and further education in 
senior secondary schooi. Twenty-two subjects are offered for this examination. Candidates must 
pass at ieast six of them inciuding Engiish to quaiify for a certificate and proceed to Form 3 (1 1th 
grade). The examination resuits are shown in Tabie 3. 


Table 3 
JCE results 

Year 

Entry 

#passing 

%passing 

1998 

74,122 

51,878 

70.0 

1999 

69,148 

63,133 

91.3 

2001 

82,530 

47,218 

57.21 


In 1997 the government phased out the JCE as a minimum requirement for entry into civii 
service. However, due to intense job competition, it is stiii used as a hiring criterion for some 
biue-coiiar jobs. Furthermore, a student must pass the JCE before sitting for the MSCE. The 
competition for Form 3 pieces is no ionger stiff since there are equai numbers of pieces in junior 
and senior secondary sections, and so aii who pass the JCE are automaticaiiy promoted to 
Form 3. 

MSCE 

MSCE, which is equivaient to High Schooi Dipioma in the US, is administered at the end of 
secondary education. The examination resuits are used for certification (i.e., certifying 
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successful completion of secondary education) and selection into the university and other 
tertiary institutions. A total of 21 subjects are offered at this level. Each subject is graded on a 
nine-point scale using the following standards: 

Grades 1-2 for distinction; 

3-6 for credit; 

7-8 for general pass; and 
9 for fail. 

To qualify for a certificate a candidate must pass at least any combination of six subjects 
including English, and one of the grades must at least be a credit pass. An MSCE certificate can 
also be awarded if a candidate passes five subjects including English, and three of which are at 
least credit passes. 

The grading process for the MSCE makes the following assumptions: 

• The examinations are equivalent across years in terms of difficulty level, content covered, 
and skills examined; 

• The test administration conditions are uniform from year to year; 

• The student cohorts taking the examination each year are randomly equivalent. 

For candidates to be considered for selection into the university, they must have earned credit or 
distinction on at least six exams, one of which must be English. A pass with at least credit grade 
in English ensures that the candidates have adequate communication skills to fully participate in 
college lectures. Currently, the University of Malawi admits only 0.3% of the secondary school 
leavers, which illustrates the stiff competition. In addition, the MSCE certificate has become the 
minimum qualification for gainful employment. Because of these two functions - selection and 
certification - there is a lot of pressure on the students to pass the examination, making it 
extremely high-stakes. 

Over the last decade there have been declining trends in pass rates on the MSCE as shown in 
Table 4. This trend has been attributed to several factors. One factor is indiscipline due to some 
student’s misinterpretation of their newly found democracy, human rights, and freedom 
(Malunga, et al. 2000). For example, Kuthemba-Mwale, Hauya and Tizifa (1996) observed “a 
general lack of interest among students to do academic school work.” Other factors that 
contribute to poor examination results include increasing number of students without a 
corresponding increase in instructional resources, and inadequate and under-qualified teachers. 
As Malunga et al. reported, there are only 4,998 secondary school teachers instead of 12,000 
required by the system. In addition, it was also observed that 67.2% of the teachers were under- 
qualified. 


Table 4 

MSCE Pass Rates 1992-99 

Year 

MSCE Entry 

#passing 

Pass (%) 

1992 

10753 

5653 

44.4 

1993 

13254 

7123 

46.7 

1994 

16264 

7871 

43.1 

1995* 

23219 

7421 

29.4 

1996 

24213 

8036 

30.7 

1997 

26543 

6740 

23.6 
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1998 

35438 

6329 

17.9 

1999 

36732 

5536 

14.3 

2001 

61856 

11143 

18.0 


*Private secondary schools opened in 1995, which may explain the large increase in number of 
students tested in this year. Other increases are harder to explain, but may be due in part to 
students who failed the exam in previous years sitting again for the exam. 

Practical, Political, and Psychometric Issues Confronted by MANEB 

MANEB administers the three national examinations described above besides other 
responsibilities such as development and administration of Malawi Craft Examinations and 
teacher certification examinations. In all these examinations the numbers of candidates and 
examination centers have been increasing every year. This has resulted in a number of 
administrative challenges, which we describe next. 

Dealing with Limited Resources 

From the three tables above, it is apparent that MANEB’s volume of work and expenditure 
increase every year. MANEB’s source of funding is largely government subvention, whose 
annual increase does not match the increased costs of administering examinations, especially in 
view of increasing inflation. 

The major areas of expenditure with regard to increasing number of examination centers that 
are scattered throughout the country are delivery and collection of examination materials to and 
from all centers, scoring, and invigilation (supervision of examinations in the centers). In 
explaining the delay in releasing the 2001 JCE and MSCE examination results, MANEB’s 
Executive Director made reference to inadequate funding as a major cause of some of the 
problems facing examination administration (The Nation, 2002a). For instance, the scoring 
process was disrupted by persistent strikes by the scorers, who were demanding more money 
from MANEB (The Nation, 2002b). It is now being proposed that examinations should be 
administered much earlier during the school year to allow adequate time for processing the 
results. The likely consequence of this proposal is that it will be difficult for schools to adequately 
cover the syllabi before the examinations are administered. 

As a way of reducing costs due to delivery and collection MANEB has introduced Examination 
Distribution Points from where surrounding schools come to collect examination materials for 
their schools on the daily basis. 

Examination Security Concerns 

One of the major concerns regarding security of examinations is leakage. In some centers 
examination envelopes have been intentionally opened before the specified time, and contents 
exposed for the benefit of candidates. In extreme cases the prematurely exposed examination 
papers have been duplicated and sold to the candidates. Such a practice led to the cancellation 
of the 2000 examinations, and another set of examination papers had to be developed and 
distributed. In an attempt to deal with this problem MANEB established Examination Distribution 
Centers for storage of examination materials which are guarded by police officers. 

Another area of concern is cheating. Cheating takes place in many forms including 
impersonation, giving extra time, substitution where a candidate’s script is replaced by one 
prepared by a more competent person, referring to books, copying from each other, copying 
from a common source, teachers dictating answers to the class, etc. As a way of curbing 
cheating during examinations, MANEB carries out spot checks during examinations, but these 
are done to a limited extent due to shortage of personnel, vehicles, and finances. MANEB also 
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provides civic education to the generai pubiic about the dangers of examination maipractice, 
since in some cases cheating invoives the generai pubiic. Sometimes MANEB appiies sanctions 
such as nuiiification of resuits, deregistration of examination centers, withhoiding resuits, and 
prosecuting the cuiprits if examinations reguiations have been infringed. In addition, MANEB 
uses externai invigiiation system whereby a teacher from a different schooi invigiiates 
examinations. The headteacher of the schoois is the overaii supervisor of examinations at the 
schooi. MANEB aiso prints examination papers outside the country to curb possibiiity of ieakage 
originating from MANEB offices. 

Dealing with Public and Political Pressure 

MANEB works under considerabie pressure because of the high-stakes nature of its 
examinations. There are many groups that directiy influence the way in which MANEB operates. 
For exampie. The Ministry of Education, which directs aii MANEB’s activities, requires timeiy 
reiease of examination resuits so that the schooi caiendar is not disturbed. For MANEB to 
administer aii examinations and process the resuits within a singie schooi year, means that 
some exams must be administered weii before the end of the schooi year to aiiow time for 
processing the resuits. Consequentiy, the examinations are iikeiy to test materiai that has not 
yet been covered in ciasses. This causes anxiety to both the examinees and their teachers. 

Another significant probiem faced by MANEB is cash flow. MANEB’s cooperating partners such 
as invigiiators, supervisors, and scorers, want to be paid promptiy for the work they do. For 
some time MANEB has not been abie to make prompt payments due to unavaiiabiiity of funds. 
This has soured the reiationship between MANEB and its partners who sometimes wait for up to 
two years before they are paid. 

Another probiem to be deait with by MANEB is score chaiienges. When examinees and their 
guardians do not agree with the examination resuits, they request a re-scoring of the exam. 
Given that the majority of MANEB exams invoive constructed-response items that are scored 
subjectiveiy, score chaiienges and re-scoring of exams is time-consuming and expensive. 

Like many educationai testing programs in the United States, MANEB is aiso a target for 
criticism in the popuiar press. Quite often, the press reports on the tension between MANEB and 
its cooperating partners, and they often highiight the negative aspects. For exampie, 
commenting on the deiay in reieasing the 2001 examination resuits. The Nation newspaper 
reported: 

MANEB and its parent ministry shouid take responsibiiity for the inconvenience that has been 
created and take necessary remediai action. Is it reaiiy impossibie to conduct incident-free 
examinations whose resuits are reieased in good time? We beiieve it is possibie and MANEB 
can oniy justify its existence by doing no iess. (The Nation, 2002a) 

In 1999, the negative coverage of examination resuits in the press prompted the State President 
to institute a commission of inquiry into the causes of poor MSCE resuits (Maiunga et ai. 2000). 
The opposition parties took advantage of the poor resuits to criticize government education 
poiicies. 

Measurement Issues 

Curricular Validity and Teaching to the Test 

The examinations in Maiawi are so important that they have assumed a “gate-keeping” roie in 
the system. Because of this importance, the examinations exert considerabie influence on what 
goes in schoois. Aithough the curricuium has generaiiy incorporated issues of the cognitive, 
psychomotor, and affective domains, examinations mainiy focus on the cognitive domain. With 
so much emphasis on passing examinations it is not surprising that the instruction has become 
examination oriented. Thus, curricuiar vaiidity of Maiawian exams is a contentious issue. 
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MANEB is aware of the demands of the curriculum, but is unable to meet them because of 
inadequate resources. In some subjects the number of examination papers was reduced and 
practical-work in some subjects was scaled down to cut down costs. For example, assessment 
by project method (Note 1) had to be discontinued. This resulted in a mismatch between the 
examinations and course objectives because only selected parts of the curriculum are 
assessed, and therefore taught. 

When projects were removed from assessment to cut down costs on project inspection and 
scoring, the schools no longer felt the need to teach by project method, even though it remained 
an important part of the national curriculum. If teachers are to cover the whole curriculum, then 
examinations must cover the curriculum. By not covering some parts of the curriculum, the 
examinations limit the scope of instruction. 

Scoring Free-Response Items 

Most MANEB exams use free-response items. Because of the large numbers of candidates the 
items are scored only once, with 10% re-scored by the Chief Examiners who supervise the 
scoring exercise. This raises the problem of examination reliability. For each of the three major 
exams, over 800,000 student papers are scored. Scoring each exam takes about four weeks 
and involves considerable human and monetary resources. 

The move by MANEB towards objective assessment using multiple-choice examinations was 
met with strong resistance from the general public who felt that the multiple-choice examinations 
would dilute the education quality. As a way of improving the reliability of the examination 
scores, MANEB put in place a number of measures such as training of scorers, pre-scoring 
exercise, standardization of scoring, script checking, and data entry verification. All these 
measures are meant to ensure that no errors are made during scoring of scripts and processing 
of examination results. However, even with this rigorous error-searching process, some errors 
go undetected and are discovered at the re-scoring stage, and only if such a request is made. 
The candidates’ requests for re-scoring are attended to only on payment of a re-scoring fee. 

High-Stakes Educational Tests in Malawi and the U.S.: Similarities and 
Differences 

The preceding sections outlined the major issues confronted by measurement professionals in 
Malawi. Interestingly, most of these issues are policy-oriented ordeal with the practical 
problems involved in test administration. Many of these issues are also confronted by 
measurement professionals in the U.S., but U.S. psychometricians appear to be more focused 
on technical issues, particularly those related to the reliability and validity of test scores. In this 
section, we discuss the similarities and differences between testing in Malawi and the U.S. with 
respect to both psychometric and educational policy issues. 

Testing and Educational Reform: An Important Area of Commonality 

It is interesting to note that educational reform movements in both Malawi and the U.S. use 
standardized tests as the primary mechanism for accountability and certification goals. Almost 
all states within the U.S. have a state-mandated assessment system (Linn, 2000), which is used 
to evaluate school districts, schools, teachers, and students. In many states, such as 
Massachusetts, state-mandated tests are also used (a) to encourage teachers to align their 
instruction with state curriculum frameworks and (b) for certification functions such as granting 
high school diplomas. 

The Malawi national examination system has also been at the heart of its educational reform 
movement. For many schools where instructional resources are scarce or nonexistent, the 
syllabi associated with MANEB tests represent significant instructional resources for teachers. 
However, it is interesting to note that the reform movement in Malawi is a national movement, 
instituted by national laws, and the tests are developed and administered by a national testing 
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agency. This situation is quite different than the U.S., where efforts to create nationaiiy 
mandated tests continuaiiy faii. States want the authority to decide what is taught and what is 
tested and so even efforts to institute the Voiuntary Nationai Test have been met with 
resistance. Oniy tests associated with the National Assessment of Educational Progress (NAEP) 
have been accepted by the states, perhaps because no student-level data are reported, the 
effect of these tests on state curricula is minimal, and there are absolutely no stakes at all for the 
students who take them. Thus, although both countries use tests as the primary data source in 
their educational accountability and certification systems, the difference between “local” and 
national control is striking. 

It is also interesting to note that “teaching to the test” is seen as a significant issue in both 
countries. In both Malawi and the U.S., there are critics who see mandated testing as a 
weakening of the curriculum, while others praise this practice as an effective means for 
improving instruction. It appears that the use of high stakes tests to improve classroom 
instruction has supporters and detractors on both continents. 

High Stakes Versus Really High Stakes 

High stakes testing receives a great deal of attention in the popular press and educational policy 
journals within the U.S. The two most common issues are the appropriateness of admissions 
tests for making postsecondary admissions decisions and the appropriateness of using 
standardized tests for awarding high school diplomas. Relatively poor performance on the SAT 
or ACT can certainly inhibit a student’s chances of getting accepted into a postsecondary 
institution, particularly the institution of her or his choice. Also, not receiving a high school 
diploma due to failing an exit exam also has serious, negative consequences for students in the 
U.S. But the stakes associated with the PSLCE and the MSCE in Malawi are much higher. With 
respect to postsecondary admissions in the U.S., the community college system is available to 
students who cannot get into a four-year college and most of these schools have open 
enrollment policies that do not require admissions test scores. More importantly, there is not a 
huge discrepancy between the number of seats available for postsecondary education and the 
number of students who seek it. With respect to high school graduation and postsecondary 
admission, the U.S. offers a multitude of well-paying jobs that do not require high school or 
college degrees. Furthermore, second-chance programs such as the Tests of General 
Educational Development and those found in adult basic education provide opportunities for 
adults who did not complete high school to earn a high school diploma later in life and continue 
their education. 

The situation in Malawi is very different. Students who do not pass the PSLCE do not even 
make it into secondary school. Even for those who do pass, there are limited spaces in the 
national secondary schools and CDSSs. Last year, only about 5% of primary school students 
were placed into the coveted national secondary schools and only 20% more were placed into 
the lower quality CDSSs. Most of the other 75% of students will never have the opportunity to 
pass the JCE and MSCE and be able to compete for the best jobs in Malawi. For many 
Malawians, passing the JCE and the MSCE makes the difference between a life of self- 
sufficiency and a life of poverty. Passing the MSCE makes numerous career options possible 
that cannot be attained through other routes. For example, the national government requires an 
MSCE certificate for civil service employment. Therefore, “high-stakes testing” has a more 
pronounced meaning in Malawi. The national educational tests are the sole criteria for academic 
certification and the stakes associated with the tests include starvation versus prosperity. 

Equity Issues in Assessment 

In the U.S., the equity issues associated with educational testing most commonly involve 
ensuring or evaluating test fairness with respect to (a) racial, ethnic, or linguistic minority groups; 
(b) females and males; and (c) individuals with disabilities. In Malawi, only sex differences in test 
performance receive significant attention by researchers, politicians, and the popular press. 


http:/ / epaa.asu.edu/ epaa/vl2n29/ 


6/28/2004 



EPAA Vol. 12 No. 29 Chakwera, Khembo & Sired: High-Stakes Testing in the Warm Hea... Page 10 of 17 


At first, an outsider may think equity issues associated with ethnicity are not reievant to Maiawi 
because aii citizens are African. However, aithough the ethnic composition of the country is 
much more homogeneous than the U.S., there are stiii significant differences with respect to 
tribai origins, reiigion, and ianguage. However, test bias with respect to these groups has not 
been extensiveiy studied. Systematic study of differences across iinguistic groups aiso has not 
been conducted, which is unfortunate since most Maiawians primariiy speak Chichewa, even 
though Engiish is its officiai ianguage (two other ianguages, Tumbuka and Yao, are aiso the 
native tongue for hundreds of thousands of Maiawians). 

The issue of accommodating tests for individuais with disabiiities has received much iess 
attention in Maiawi than in the U.S. There is no acknowiedgement of students with iearning 
disabiiities in the educationai system and so granting extended time on tests to such students, 
which is common in the U.S., is not even on the radar screen. However, MANEB does make 
Braiiie tests avaiiabie to students with visuai disabiiities and provides 1/6 additionai time for such 
students to take the tests. 

In the U.S., equity issues are at the forefront of educationai assessment poiicy debates. When 
achievement differences are found across raciai/ethnic groups on educational tests, 
researchers, lawmakers, and policy analysts are often divided about what should be done. 
Claims of test bias against minority groups have led to the abandonment of some educational 
tests, but affirmative action practices (e.g., using different standards for selecting minority and 
non-minority candidates) have not stood up to legal scrutiny (Green & Sired, 1999; Sired & 
Green, 2000). These policy issues have led psychometricians and other researchers to focus 
much of their research on issues of adverse impact and test bias. For example, studies of 
differential predictive validity and differential item functioning are common in the U.S., but are 
practically non-existent in Malawi. 

An interesting difference between Malawi and the U.S. with respect to equity in assessment is 
the way they handle sex differences on educational tests. In the U.S., within-group norming 
(Note 2) practices have been outlawed for organizations that receive federal funds, which 
include virtually all accredited educational institutions and all governmental agencies. Thus, 
adjusting for performance differences across males and females is not conducted. In Malawi, 
performance differences between females and males on educational tests are more 
pronounced. Furthermore, the proportions of females at secondary and postsecondary schools 
are well below that of males. Thus, colleges and universities struggle to admit qualified females. 
To address educational opportunity differences across the sexes, Malawi secondary schools, 
colleges, and universities rank females and males separately so that the highest-ranking 
females will be accepted over males that may have scored higher on a test. Thus, given the 
same equity issue, the two countries made completely opposite policy decisions. This difference 
stems not so much from philosophical differences in assessment or admissions equity, but from 
differences in the numbers of women remaining in school after the primary grades. 

Use of Item Formats 

As described above, MANEB exams use predominantly constructed-response items. Multiple- 
choice items are used on some exams, but the public perception is that such items dumb down 
the curriculum and are not effective for measuring important academic knowledge and skills. 
These criticisms have also been raised in the U.S., but the psychometric community has worked 
hard to educate the public about the benefits of multiple-choice items (e.g., increasing score 
reliability and content coverage, measuring higher-level skills, reduced scoring costs and 
reduced testing time) as well as the limitations of constructed-response items (Note 3) (lack of 
content coverage, task specificity, reduced reliability, higher scoring costs). In the U.S., the 
majority of educational tests use either only multiple-choice items, or a combination of multiple- 
choice and constructed response items. These practices reflect a desire to ensure adequate 
levels of score reliability and content validity while keeping down scoring costs. In Malawi, 
construct representation is emphasized at the expense of score reliability, testing time, score 
reporting time, and scoring costs. 
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Computer-Based Testing 

Another striking difference between educationai assessment in Maiawi and the U.S. is the 
amount of attention paid to computer-based testing (CBT). in the U.S. aimost aii testing 
programs are moving towards computerized administration of their tests or are considering the 
use of computers in improving their assessment systems (Zenisky & Sireci, 2002). Conferences 
within the educationai measurement community feature programs that are dominated with CBT 
issues such as computerized-adaptive testing, innovative item types, and automated scoring of 
constructed-response items. These topics are not receiving considerabie attention in Maiawi, 
primariiy due to the iack of computer resources within the country. 

Measurement Community 

Another huge difference between the U.S. and Maiawi is the presence of a significant 
educationai measurement community, in the U.S. there are thousands of measurement 
professionais who meet and interact reguiariy. For exampie, there are approximateiy 3,000 
members of the Nationai Councii on Measurement in Education (NCME) and even more 
members of the Measurement, Evaiuation, and Statistics Division of the American Educationai 
Research Association (AERA). The Psychometric Society and the measurement and statistics 
division of the American Psychoiogicai Association (APA) aiso provide nationai forums for 
measurement professionais. in Maiawi, the measurement community is much younger and 
much smaiier. For exampie, there is no Maiawi equivaient of the Standards for Educational and 
Psychological Testing (AERA, APA, NCME, 1999). However, in 2001, a grant from USAiD to the 
University of Massachusetts Amherst (UMASS) estabiished a program to buiid educationai 
measurement expertise within Maiawi. Currentiy, nine measurement professionais from Maiawi 
are receiving doctorai or master’s degrees in psychometrics from UMASS and an educationai 
measurement program is being reinforced at the University of Maiawi. This deveiopment shouid 
bring measurement practices and research areas across the two countries cioser together in the 
future. 

Technical Versus Practical Measurement Issues 

In addition to differences in the attention paid to differential predictive validity and differential 
item functioning, there are also significant differences between Malawi and the U.S. with respect 
to the issues that receive the most attention as part of the normal operating procedures of 
testing agencies. For example, in the U.S., procedures for scaling educational tests are widely 
researched and item response theory (IRT) is a common procedure for scaling educational 
tests. Furthermore, tests administered in different years are typically equated onto a common 
scale to ensure differences in test difficulty are taken into account when monitoring student 
progress and awarding credentials. In Malawi, IRT is not used at all, and tests are not equated 
across years. Instead, different test forms are assumed to be equivalent in content and difficulty. 
Another significant difference is in procedures used to set standards on educational tests. In the 
U.S., standard setting is one of the busiest areas of research and new methods appear 
continuously (e.g., Cizek, 2000). In Malawi, standard setting is conducted in a less systematic 
fashion drawing from subjective estimates of test difficulty and student cohort differences. 

Due to the higher stakes, more limited resources, and limited technical expertise, the 
measurement issues that get the most attention in Malawi are more logistical. Reducing 
cheating is a significant issue, since it is widespread and it represents a significant threat to the 
validity of exam scores. Developing, administering, and scoring the exams essentially exhausts 
the personnel and financial resources of MANEB and so there is little time or resources to 
conduct research on test validation. 

Conclusions: Testing Collegiality Around the Worid 

This paper illustrates how different countries deal with common measurement issues, as well as 
those that are unique to their own situation. Many of the practical problems in measurement are 
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universal and so much can be learned from what other countries are doing. For example, the 
U.S. can learn from Malawi about successful implementation of large-scale performance 
assessment and about alternative strategies for achieving equity in test-based admissions 
decisions. Malawi can learn technical measurement solutions to problems such as scaling, 
equating, standard setting, and item and test bias research. 

By building international collegiality within the measurement community we will be better 
positioned to help each other tackle our significant measurement problems. For example, there 
is much that could be done in both countries to build computerized systems for test delivery that 
could reduce cheating and test administration costs. Also, measurement programs in the U.S. 
could do more to reach out and train professionals in Malawi. This expertise could then be 
extended to other countries in Africa through the measurement program at the University of 
Malawi and through similar programs that could be developed in other countries. Quality 
educational systems need quality assessments. Through the process of building measurement 
expertise in developing countries such as Malawi, we can help these countries improve their 
educational systems. 

Notes 

This research was entirely collaborative and the order of the authors is alphabetical. 

Correspondence concerning this article should be addressed to Stephen G. Sired, Center for 
Educational Assessment, School of Education, University of Massachusetts, Amherst, MA 
01003-4140. E-mail correspondence may be sent to Sireci@acad.umass.edu. 

1 . Assessment by project refers to embedded assessments where students complete hands-on 
projects throughout the school year that were graded by MANEB. One assessment that was 
cancelled was an agricultural project where MANEB officials visited farm sites and evaluated 
students’ agricultural projects. 

2. In within-group norming, candidates within a group (say, male or female) are rank-ordered 
with respect to everyone else in the group. Then, the ranks of the candidates are treated as if 
they were interchangeable. For example, the highest-ranking female would be considered 
equivalent to the highest-ranking male, even if their scores on a test were very different. 

3. See Dunbar, Koretz, & Hoover, 1991; Linn & Burton, 1994; Wainer & Thissen, 1998, for 
empirical studies of the advantages and disadvantages of these different item formats. 
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